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Abstract 


Unmanned  Combat  Air  Vehicles  extend  tactical  the  range  of  aircraft  mission  options.  Although 
certain  cultural  barriers  exist  with  regard  to  complete  acceptance  of  remotely  controlled  aircraft,  the 
UCAV  will  become  an  essential  part  of  many  Air  Forces  in  the  years  to  come.  A  critical  factor  to 
the  success  of  UCAV  missions  is  the  optimal  selection  of  operators.  This  paper  explores  options  for 
improving  the  selection  and  training  of  UCAV  operators  and  reviews  test  that  may  be  useful  toward 
an  improved  screening  system.  Emphasis  is  placed  upon  the  assessment  of  decision-making  ability, 
personality  factors,  and  related  cognitive  attributes  with  recommendations  for  future  research 
directions.  In  the  interest  of  identifying  a  valid  UCAV  test  battery  that  will  reliably  predict 
successful  completion  of  both  UCAV  training  and  operational  performance,  program  managers  are 
encouraged  to  consider  funding  selection  programs  that  evaluate  psychological  measures  of 
individual  differences,  in  addition  to  the  standard  psychomotor  coordination  measures. 


*The  authors  gratefully  acknowledge  the  significant  assistance  and  contributions  to  this  paper  by 
LCDR  Sean  Biggerstaff  of  the  Naval  Operational  Medical  Institute. 
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Introduction 

Uninhabited  combat  air  vehicles  (UCAVs)  are  predicted  to  dominate  the  battlespace  in  the 
coming  century.  They  will  be  intelligent,  autonomous,  systems  of  systems.  UCAVs  are  envisioned 
to  conduct  a  myriad  of  missions  including:  surveillance  and  reconnaissance,  precision  strike, 
suppression  of  enemy  air  defense  (SEAD)  and  several  logistical  (resupply)  efforts.  Military'  planners 
expect  the  UCAV  to  be  a  transparent  extension  of  existing  manned  combat  systems.  This  goal  will 
only  be  attained  though  effective  operator  selection  and  crew  systems  integration  (CSI). 

Very  little  research  has  been  conducted  to  assess  the  operator  selection  and  crew  systems 
integration  (CSI)  requirements  essential  to  understanding  the  human  limitations  necessary  for 
optimizing  the  interfaces  of  these  complex  systems.  Current  concepts  of  uninhabited  air  vehicle 
control  are  based  on  the  remotely  placed  operator  as  a  controller  of  the  vehicle's  flight  and  require 
several  crewmembers  to  control/manage  a  single  air  vehicle.  Future  concepts  envision  the  remote 
operator  as  more  of  a  “mission  manager  .  providing  primarily  operational  mission  information  to 
the  vehicle,  including  information  concerning  target  acquisition  and  weapons  release,  and  mav 
require  a  single  operator  to  effectively  control/manage  several  air  vehicles  simultaneouslv.  In  order 
to  achieve  force  multiplication  using  UAVs.  yet  reduce  the  number  of  dedicated  personnel  afloat  or 
aloft  the  ability  for  a  single  operator  to  simultaneously  control/manage  more  than  one  of  these 
vehicles  is  necessary. 

UCAVs  are  going  to  be  a  fourth  class  of  vehicle  that  will  contain  many  of  the  same 
operator/system  problems  that  have  existed  for  years  in  the  other  three  (manned  aviation,  cruise 
missiles,  and  UAVs).  However,  it  will  also  create  many  issues  that  haven't  been  addressed  in  prior 


systems. 
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Greater  automation  of  the  vehicle  system  will  allow  the  operator/mission  manager  to  focus 
attention  on  the  mission  rather  than  on  tasks  associated  with  vehicle  flight  as  is  typically  the  case 
with  the  existing  legacy  UAVs.  Current  systems  rely  upon  human  intervention  to  control  the  air 
vehicle.  The  US  Navy's  Pioneer  U  AV  requires  an  external  pilot  to  manually  control  the  flight 
surfaces  of  the  vehicle  during  takeoff  and  landing  through  an  interface  derived  from  that  of  a  remote 
controlled  airplane.  Physical  dexterity  and  hand-eye  coordination  are  taxed  through  the  use  of  this 
system.  The  US  Air  Force's  Predator  UAV  requires  control  by  a  rated  pilot.  Rudder  pedals,  throttle 
and  joystick  are  housed  in  the  ground  control  station.  Once  again,  physical  dexterity  is  required. 

UCAVs  are  envisioned  as  the  next  generation  of ''smart  UAVs’\  which  will  place  greater 
cognitive  demands  on  the  operator/mission  manager.  UCAVs  are  envisioned  to  be  autonomous  with 
little  or  no  manual  control  required.  Performance  of  tasks  that  are  likely  to  be  required  include: 

managing/controlling  multiple  air  vehicles*  missions:  recognizing  and  dealing  with 
degraded  system  functionality;  regaining  situational  awareness  after  loss  of  vehicle  data 
links:  interpreting  displays  containing  multiple  UCAV  perspectives:  and  shift  of  system 
control  to  other  team  members  or  control  stations. 

It  should  be  apparent  then,  that  the  emphasis  will  be  on  the  operator/manager's  cognitive  ability  to 
assimilate  and/or  disseminate  dynamic,  complex  data  in  order  to  make  rapid  and  effective 
strike/attack  decisions.  The  role  of  psychomotor  abilities  in  UCAV  operations  is  likely  to  be 
diminished. 

Selection  of  operators  for  UCAV  missions  will  have  to  consider  abilities  in  the  cognitive  as  well  as 
psychomotor  domains.  Additionally,  questions  that  must  be  considered  include  the  following: 
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Is  the  operator  a  pilot  or  non-pilot? 

What  are  the  skill  requirements  (performance  requirements)? 

What  should  be  the  team  composition?;  and 

What  aspects  of  previous  training  will  have  positive  and/or  negative  impacts  on 
transfer  of  training? 

UAV  Operator  Selection 

In  response  to  a  series  of  costly  Pioneer  UAV  craft  losses  attributed  to  human  error. 
Biggerstaff.  et  al  conducted  a  UAV  operator  task  analysis.  As  a  result  of  the  study,  both  medical  and 
operator  selection  test  recommendations  were  proposed  ( I ). 

Several  performance-based  selection  test  measures  were  identified  and  evaluated  in  the  study  to 
predict  UAV  operator  training  outcome.  Although  the  sample  size  in  the  study  was  small 
(N=I4  ),  significant  findings  were  reported  between  a  composite  of  multitask  tracking  scores  and 
UAV  performance.  The  following  is  a  brief  summary  of  the  tests  used  in  the  study  ( I ).  Detailed 
descriptions  of  the  tests  can  be  found  in  Helton.  Nontasak.  and  Dolgin.  (2). 

Psychomotor  Task  (PMT).  The  PMT  consists  of  stick,  rudder  pedal,  and  throttle  control  tasks  and 
measures  eye,  hand,  foot  coordination. 


Horizontal  Tracking  (HT).  In  this  task,  the  subject  is  required  to  learn  a  one-dimensional 


compensatory  tracking  task. 


Dichotic  Listening  Task  (OLT).  The  DLT  assesses  individual  differences  in  the  ability  to  focus 


attention 
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PMT/DLT  Combination.  As  a  measure  of  time-sharing  abilities,  the  PMT  can  be  combined  with  the 
DLT  in  order  to  assess  shared  resources.  As  mentioned  above,  the  PMT  and  DLT  are  administered 
initially  as  single  tasks  and  then  combined  to  result  in  increasingly  complex,  simultaneously 
performed,  multiple-task  conditions. 

Manikin  Test.  The  Manikin  Test  measures  spatial  orientation  and  reaction  time. 

Digit  Cancellation  Task.  The  subject  is  visually  presented  with  a  random  number  between  I  and  9 
on  the  computer  screen  that  is  erased  and  then  followed  immediately  with  another  number.  Although 
this  task  is  essentially  a  measure  of  short-term  memory  and  reaction  time,  it  can  also  be  used  as  a 
distractor. 

Digit  Cancellation  and  Horizontal  Tracking  The  horizontal  tracking  and  digit  cancellation  tasks  are 
combined  for  a  measure  of  dual-task  performance. 

Time  Estimation  Task.  The  Time  Estimation  Task  measures  v  isual  tracking  and  time  estimation 
abilities  The  horizontal  tracking  task  is  included  as  part  of  this  task  as  a  distractor.  According  to 
Biggersiaff  et  al  (1998).  the  task  measures  of  time  and  distance  estimation  abilities  are  critical  to  the 
external  pilot’s  control  of  the  Pioneer. 

.Although  limited  to  psychomotor/spatial  testing,  the  test  battery  identified  in  the  Naval 
Aerospace  Medical  Research  Laboratory  study  is  a  good  first  step  toward  establishing  UCAV 
operator  assessment  and  selection. 
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Psychomotor  and  spatial  abilities-type  tests  have  been  used  (or  proposed  for  use)  in  one 
form  or  another  with  different  populations  since  World  War  I  (3).  They  have  successfullv  been  used 
to  predict  training  outcome  and  have  been  shown  to  account  for  significant  variance.  In  particular, 
psychomotor  coordination  tests  have  been  demonstrated  to  reliably  account  for  significant  amounts 
of  variance  in  the  prediction  of  flight  training  success  (4).  However,  as  operational  requirements  and 
technology  have  become  more  complex  and  demanding,  variables  such  as  basic  reaction  time, 
spatial  ability  and  eye-hand-foot  coordination  become  less  task  relevant. 


While  the  relevance  of  psychomotor  variables  has  declined,  there  has  been  an  increase  in  the 
relevance  of  high  level  cognitive  functions  for  UCAV  operations.  As  previously  described.  UCAVs 
place  tremendous  demands  on  multi-tasking,  prioritizing,  sequencing,  and  vigilance.  Unlike  the 
psychomotor  performance  area,  where  there  are  numerous  tests  available  for  measuring  abilities,  there 
are  relatively  few  tools  designed  for  measuring  multi-tasking,  prioritizing  and  sequencing.  Computer- 
based  cognitive  testing,  including  virtual  reality-based  testing  approaches,  appears  to  be  the  most 
promising  approach  for  assessing  the  abilities  of  UCAV  candidates  and  incumbent  operators. 
Computerization  provides  for  excellent  standardization  of  test  stimuli,  time  intervals,  maximal  control 
over  item  presentation,  capacity  to  present  degraded  stimuli,  suitability  for  presentation  of 
simultaneous  mental  tasks  (in  the  same  or  different  sensory  modalities),  and  accurate  recording  of 
response  speeds  and  accuracy  to  all  tasks  being  presented. 

CogScreen-Aeromedical  Edition  is  a  computerized  cognitive  screening  test  originallv  designed  for 
the  U.S.  Federal  Aviation  Administration  as  an  instrument  for  evaluating  pilots'  neurocognitive 
fitness-to  fly.  The  selection  of  the  tests  for  CogScreen-AE  was  based  on  existing  task  anaivsis  of 
the  cognitive  and  psychomotor  demands  of  flying.  In  fact  much  of  the  pioneer  work  conducted  bv 
the  Walter  Reed  Army  Institute  of  Research  and  the  Navy  Aerospace  Medical  Research 
Laboratories  was  incorporated  into  the  development  of  CogScreen-AE.  Studies  have  demonstrated 
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that  CogScreen-AE  is  not  only  sensitive  to  changes  in  brain  functioning  resulting  from  trauma, 
substance  abuse  or  illness,  but  that  it  is  also  a  strong  predictor  of  flight  performance. 

In  one  study,  the  frequency  and  severity  of  flight  performance  violations  for  a  substantial  group  of 
captains  was  measured  by  flight  data  recorders.  Specifically,  Yakimovich,  Strongin,  Govorushenko. 
Schroeder,  and  Kay  presented  a  paper  at  the  65*'^  Annual  Scientific  Meeting  of  the  Aerospace 
Medical  Association  in  1994.  The  investigators  reported  that  variables  from  a  small 
number  of  CogScreen-AE  tests,  partciularly  those  measuring  working  memory,  mental  flexibility, 
and  divided  attention  (e.g..  Dual  Task,  Shifting  Attention,  Divided  Attention,  Pathfinder,  and 
backward  Digit  Span),  were  able  to  account  for  between  30%  and  45%  of  the  varianice  in  the 
perfonnance  index,  e.g..  flight  violations  per  100  hours  of  flight.  Hoffmann  et  al.  found  a  strong 
relationship  between  these  measures  and  performance  in  the  cockpit,  including  CRM  (5). 
Encouraging  results  from  these  and  other  studies  has  led  the  US  Air  Force  to  include  CogScreen-AE 
as  a  component  of  their  Enhanced  Flight  Screen  for  aviators  (6).  Subsequently  a  large  number  of 
commercial  airlines  in  the  US  and  elsewhere  began  using  CogScreen-AE  for  pilot  selection  and  to 
establish  a  baseline  against  which  to  evaluate  their  cognitive  functioning  at  some  later  time. 

UCAV:  Cognitive  Testing 

While  the  relevance  of  psychomotor  variables  has  declined  in  pilot  selection,  there  has  been  an 
increase  in  the  relevance  of  high  level  cognitive  functions.  As  described  previously,  UCAV 
operations  place  tremendous  demands  on  multi-tasking,  prioritizing,  sequencing,  and  vigilance. 
Unlike  the  psychomotor  performance  area,  where  there  are  numerous  tests  available  for  measuring 
abilities,  there  are  relatively  few  tools  designed  for  measuring  these  higher  level  cognitive  abilities. 
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Computer-based  cognitive  testing,  including  virtual  reality- based  testing  approaches, 
appears  to  be  the  most  promising  approach  for  assessing  the  abilities  of  UCAV  candidates  and 
incumbent  operators.  Computer-based  testing  provides: 

•  excellent  standardization  of  test  stimuli  and  time  intervals; 

•  offers  maximal  control  over  item  presentation: 

•  capacity  for  presenting  degraded  stimuli; 

•  suitability  for  presenting  simultaneous  mental  tasks  (in  the  same  or  different  sensorv 
modalities);  and 

•  accurate  recording  of  response  speeds  and  response  accuracy. 


CogScreen-Aeromedical  Edition  is  a  computerized  cognitive  screening  test  originally  designed  for 
the  U.S.  Federal  Aviation  Administration  (FAA)  as  an  instrument  for  evaluating  pilots' 
neurocognitive  fitness-to  fly.  The  selection  of  the  tests  for  CogScreen-AE  was  based  on  existing  task 
analysis  of  the  cognitive  and  psychomotor  demands  of  flying.  In  fact  much  of  the  pioneer  work 
conducted  by  the  Walter  Reed  Army  Institute  of  Research  and  the  Navy  Aerospace  Medical 
Research  Laboratories  was  incorporated  into  the  development  of  CogScreen-AE.  Studies  have 
demonstrated  that  CogScreen-AE  is  not  only  sensitive  to  changes  in  brain,  functioning  resultina 
from  trauma,  substance  abuse  and  illness,  but  that  it  is  also  a  strong  predictor  of  flight  perfonnance. 


UCAV;  Personality^  Testing 

CogScreen-AE,  and  similar  computer-based  tests,  may  be  helpful  in  assessing  the  cognitive  abilities 
relevant  to  UCAV  operations,  however,  operator  selection  also  needs  to  address  the  personality 
characteristics  and  relational  attributes  of  prospective  operators.  In  a  stressful  and 
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demanding  environment,  the  importance  of  interactional  style,  judgement  and  decision-making  can 
make  the  difference  between  a  successfully  completed  mission  and  the  loss  of  a  million  dollar 
resource.  In  fact,  it  is  precisely  these  measures  of  individual  differences  in  personality  that  predict 
outcomes  in  operational  tasks. 

Efforts  to  improve  personality  assessment  include  computerization,  development  of  verification 
and  correction  scales,  keying  certain  items  against  specific  criteria,  masking  the  dimension  of 
interest,  and  the  application  of  factor  analysis  to  isolate  more  specific  trait  categories  (7.  8.  9). 
Historically,  there  have  been  validity  problems  and  lackluster  results  associated  with  the  use  of 
personality  tests  in  selections.  This  is  largely  due  to  the  obvious  psychopathology  orientation  of  the 
tests  used  in  past  personnel  selection  research. 

Commercial  airlines  have  used  clinical  personality  tests  as  well  as  interviews  for  pilot  selection. 
Meta-analyses  and  related  research  provide  compelling  evidence  that  personality  traits  are  predictive 
ot  training  and  Job  performance  across  a  wide  range  of  occupational  settings  (10.  11.  12).  In  fact,  it 
has  been  demonstrated  that  personality  profiles  differ  with 

USAF  pilot  role:  fighter,  bomber,  tanker  (13). 

There  are  a  number  of  personality  tests  that  may  prove  useful  in  UCAV  operator  selection.  Two 
examples  are  as  follows: 

Revised  NEO  Personality  lnventor\': 

The  NEO  represents  a  new  approach  to  measuring  the  Big  Five  personality  measures.  It  is  based  on 
extensive  research  that  shows  that  a  consistent  set  of  dimensions  of  normal 
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personality  across  the  psychological  literature.  These  are  referred  to  tliese  as  Neuroticism. 
Openness.  Agreeableness.  Conscientiousness  and  Extroversion  (14).  A  significant 
correlation  was  found  between  agreeableness  and  perfonnance  on  measures  of  crew 
resource  management  (CRM)  (5). 


FIRO-B: 

The  Fundamental  Interpersonal  Relation  Orientation  (FIRO-B)  generates  6  scores  (0-9) 
indicating  expressed  and  wanted  needs  in  the  areas  of  Inclusion.  Control,  and  Affection. 
Scores  from  0  to  I  are  considered  e.xtremely  low  and  scores  from  8  to  9  are  considered 
extremely  high  scores.  According  to  Schutz.  the  basic  idea  is  that  "everv  person  orients 
himself  (or  herself)  in  characteristic  ways  toward  other  people,  and  that  knowledge  of  these 
orientations  allows  for  considerable  understanding  of  individual  behavior  and  the 
interaction  of  people"  (1 5).  In  another  study,  "group  warmth",  a  derivative  of  FIRO-B 
Inclusion  and  Affection  scales,  was  significantly  related  to  the  commercial  effectix  eness  of 
teams  (16).  The  FIRO-B  was  also  evaluated  as  a  measure  of  team  compatiblv  with  U.S.  Air 
Force  AWACS  weapons  director  teams  ( 1 7).  Although  their  results,  in  general,  did  not 
support  the  hypothesis  that  high  compatibility,  as  measured  by  the  FIRO-B.  would  be 
associated  with  better  simulated  AWACS  performance,  crew  compatability  was  a  factor  in 
team  performance  for  several  of  the  teams  in  the  study.  .According  to  Shutz.  the  FIROB  has 
been  successfully  used  for  "improving  self-awareness,  teamwork,  morale,  and  productivity 
in  such  organizations  a  Proctor  &  Gamble.  AT&T.  NASA.  Amdahl  Corporation,  the 
Swedish  Army  and  about  1 00  companies  in  Japan  ( 1 6). 
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UCAV:  Tests  of  Basic  Aviation  Knowledge  and  Intelligence 

Multi-tasking  ability  is  required  to  operate  under  stress  and  respond  to  multiple  stimuli.  However,  to 
perform  the  UCAV  tasks  described  in  the  background  section  of  this  paper,  the  operator  must  be 
able  to  formulate  an  idea,  make  a  correct  decision  and  respond.  This  requires  two 
additional  measures,  g.  or  basic  intelligence,  and  aviation  knowledge.  The  task  of  operating  multiple 
vehicles  at  a  distance  and  deriving  necessary  information  from  systems  and  instrumentation  is  an 
abstract  exercise  filled  with  the  interpretation  of  abstract  concepts.  Facility  with  abstract,  verbal, 
mathematical  and  spatial  reasoning  is  expected  to  be  as  important  as  it  has  been  in  aviation  ( 1 8).  It  is 
also  clear  from  the  work  of  Hoffmann  et  al.  that  knowledge  of  aerodynamics,  engineering  and 
navigation,  at  a  conceptual  level,  is  likely  to  be  predictive  of  performance  for  these  operators  (5). 
Tests  in  these  content  areas  need  to  be  constructed  to  assess  the  ability  of  the  operators  to  actually 
apply  principles  of  engineering,  and  aerodynamics  to  solve  novel  problems.  This  type  of  problem 
solving  is  quite  different  form  the  use  of  check  lists  for  troubleshooting  equipment  failures. 

Model  for  Develoning  a  Test  Battery  for  UCAV  Operator  Selection 

The  abilities  required  for  UCAV  operations  fall  into  fi\e  categories: 

g; 

psychomotor  ability; 
cognitive  ability; 
personality;  and 
conceptual  knowledge. 

An  effective  selection  test  battery  will  need  to  use  adaptations  of  existing  tests  in  these  five  areas,  or 
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use  newly  developed  tests.  These  abilities  can  be  viewed  as  orthogonal;  high  psychomotor  ability 
does  not  predict  high  conceptual  knowledge  or  even  high  cogniti\  e  ability.  Measures  in  each  of 
these  ability  areas  (used  individually  and  in  combinations)  have  been  shown  to  predict  success  in 
flight  training  and  line  operations. 

The  steps  to  develop  such  a  test  battery’  are  well  established: 

•  identify  relevant  knowledge,  skills,  abilities,  and  personality  attributes  as  well  as 
specific  performance  objectives  that  operators  are  required  to  achieve; 

•  define  the  environment  in  which  they  will  be  expected  to  work: 

•  identify  and  link  knowledge,  skills,  abilities,  and  personality  characteristics  that  are 
associated  with  success  at  those  activities  and  in  those  conditions: 

•  select  or  develop  measures  of  those  knowledge,  skills,  abilities,  and  personality 
parameters: 

•  develop  performance  measures  of  job  related  acti\  ities; 

•  test  a  representative  sample  of  individuals  believed  capable  of  learnina  and  performint; 
the  job  of  UCAV  operator,  (clearly  this  would  ha\e  to  occur  on  simulators  ahead  of 
production  of  the  UCAVs.  Note  that  such  research  would  be  informative  of  the  human 
factors  limitations  of  specific  UCAV  designs); 

•  train  operators; 

•  observe  operator  performance  in  training  and  on  the  job; 

•  correlate  performance  with  test  measures  and  build  a  mathematical  model  that  relates 
success  on  the  job.  and  success  in  training,  w  ith  performance  on  tests; 

•  choose  cut  scores  to  maximize  true  positives  and  minimize  false  positives. 
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Discussion 

In  addition  to  the  abilities  discussed  in  this  paper,  there  are  others  that  may  be  important  to 
consider  as  part  of  a  selection  assessment  battery.  For  example,  the  operator's  willingness  to  rely  on 
a  computer  for  decisions  may  be  a  particularly  important  factor  for  the  UCAV  operator  functioning 
in  a  virtual  reality  like  environment.  As  UCAV  s\’stems  evolve  it  may  become  apparent  that  the 
individual  s  style  of  interacting  as  a  member  of  a  team,  may  also  be  a  critical  selections  variable. 
Ultimately,  the  objective  is  to  develop  a  comprehensive  UCAV  test  battery  that  includes  not  only 
the  basic  psychomotor  coordination  measures  but  one  that  measures  all  of  the  relevant  personal  itv 
and  cognitive  attributes.  Development  of  a  comprehensive  assessment  battery  for  UCAV  operators 
is  likely  to  result  in  benefits  in  UCAV  system  engineering  as  well. 
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